R Notebook

Author: Dhanush Kobal

Project Name: Regression analysis

We are going to do another data analysis on house price data and see how accurate our analysis will be

Evaluation metric: \(\sqrt{\frac{1}{n} \sum_{i=1}^n (log(y_{obs}+1)-log(y_{pred}+1))^2}\)

We will assume that the train data is a good representative of the test data

Loading required package: ParamHelpers
Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     
Warning message: 'mlr' is in 'maintenance-only' mode since July 2019. Future development will only
happen in 'mlr3' (<https://mlr3.mlr-org.com>). Due to the focus on 'mlr3' there might be uncaught
bugs meanwhile in {mlr} - please consider switching.
Registered S3 methods overwritten by 'dbplyr':
  method         from
  print.tbl_lazy     
  print.tbl_sql      
── Attaching packages ───────────────────────────────────────────────────────────────────── tidyverse 1.3.1 ──
✓ ggplot2 3.3.5     ✓ purrr   0.3.4
✓ tibble  3.1.3     ✓ dplyr   1.0.7
✓ tidyr   1.1.3     ✓ stringr 1.4.0
✓ readr   2.0.0     ✓ forcats 0.5.1
── Conflicts ──────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
x dplyr::filter() masks stats::filter()
x dplyr::lag()    masks stats::lag()

Attaching package: ‘FSelectorRcpp’

The following object is masked from ‘package:FSelector’:

    relief

We will first see how the data is presented

Rows: 1460 Columns: 81
── Column specification ──────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (43): MSZoning, Street, Alley, LotShape, LandContour, Utilities, LotConfig, LandSlope, Neighborhood, C...
dbl (38): Id, MSSubClass, LotFrontage, LotArea, OverallQual, OverallCond, YearBuilt, YearRemodAdd, MasVnrA...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 1459 Columns: 80
── Column specification ──────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (43): MSZoning, Street, Alley, LotShape, LandContour, Utilities, LotConfig, LandSlope, Neighborhood, C...
dbl (37): Id, MSSubClass, LotFrontage, LotArea, OverallQual, OverallCond, YearBuilt, YearRemodAdd, MasVnrA...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 1459 Columns: 2
── Column specification ──────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
dbl (2): Id, SalePrice

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
[1] 1460
[1] 1459

Warning: It is deprecated to specify `guide = FALSE` to remove a guide. Please use `guide = "none"` instead.

We will look at all the numerical variables to first see if we need to convert them into factors

---

A preview of our updated data

---

Outlier analysis

We will see each numeric variables and see if we need to delete any observation

We will see how representative the train data is with the test data, so I will create factor that tells us which observation is a test data or train data

Outlier analysis: Lotarea

Outlier analysis: MasVnrArea

Warning: Removed 23 rows containing missing values (geom_point).

[1] 298

Warning: Removed 23 rows containing missing values (geom_point).

Outlier analysis: BsmtFinSF1

Warning: Removed 1 rows containing missing values (geom_point).

[1] 1299

Warning: Removed 1 rows containing missing values (geom_point).

Outlier analysis: X3SsnPorch

We removed 1 outlier

[1] 206

The rest of the data seems like a good representative of the test data

We are first going impute the dataset to do further analysis.

---

Data imputation

We will impute the numerical variables using the Mode and we will impute the categorical variable with a ML Algorithm (naiveBayes)

Now it is important for us to see if the levels in the test and train data are the same.

Now we will check how well the imputation went

The imputed values seems reasonable. Yes this does produce a bias in our model, however, this bias might lead to better performance

---

Feature engineering

Feature engineering: Street

Feature engineering: LotShape

ggplot(data = combined.data[train.kaggle$Id, ] , aes(x = LotShape, y = OverallQual)) + geom_bar(stat = "identity")

a<-ifelse(combined.data$LotShape=="Reg" , "good.overallQual",
          ifelse(combined.data$LotShape %in% c("IR1") , "medium.overallQual" , "bad.OverallQual"))

combined.data$LotArea.OverallQual.factor<-as.factor(a)

Feature engineering: Neighborhood

Feature engineering: OverQuall

Feature engineering: OverCond

Feature engineering: TotRmsAbvGrd

---

Quantitative feature engineering

TSNE

[1] -31.35391  19.75975 -23.86412 -28.19424 -10.59565 -10.24031

Other numerical factors that might be useful

Correlation Plot

---

We will see if are any data imbalances

$MSSubClass
x
  20   30   40   45   50   60   70   75   80   85   90  120  150  160  180  190 
0.37 0.05 0.00 0.01 0.10 0.20 0.04 0.01 0.04 0.01 0.04 0.06 0.00 0.04 0.01 0.02 

$MSZoning
x
C (all)      FV      RH      RL      RM 
   0.01    0.04    0.01    0.79    0.15 

$Street
x
Grvl Pave 
   0    1 

$LotShape
x
 IR1  IR2  IR3  Reg 
0.33 0.03 0.01 0.64 

$LandContour
x
 Bnk  HLS  Low  Lvl 
0.04 0.03 0.02 0.90 

$Utilities
x
AllPub NoSeWa 
     1      0 

$LotConfig
x
 Corner CulDSac     FR2     FR3  Inside 
   0.18    0.06    0.03    0.00    0.72 

$LandSlope
x
 Gtl  Mod  Sev 
0.95 0.04 0.01 

$Neighborhood
x
Blmngtn Blueste  BrDale BrkSide ClearCr CollgCr Crawfor Edwards Gilbert  IDOTRR MeadowV Mitchel   NAmes 
   0.01    0.00    0.01    0.04    0.02    0.10    0.04    0.07    0.05    0.03    0.01    0.03    0.15 
NoRidge NPkVill NridgHt  NWAmes OldTown  Sawyer SawyerW Somerst StoneBr   SWISU  Timber Veenker 
   0.03    0.01    0.05    0.05    0.08    0.05    0.04    0.06    0.02    0.02    0.02    0.01 

$Condition1
x
Artery  Feedr   Norm   PosA   PosN   RRAe   RRAn   RRNe   RRNn 
  0.03   0.06   0.86   0.01   0.01   0.01   0.02   0.00   0.00 

$Condition2
x
Artery  Feedr   Norm   PosA   PosN   RRAe   RRAn   RRNn 
  0.00   0.00   0.99   0.00   0.00   0.00   0.00   0.00 

$BldgType
x
  1Fam 2fmCon Duplex  Twnhs TwnhsE 
  0.84   0.02   0.04   0.03   0.08 

$HouseStyle
x
1.5Fin 1.5Unf 1Story 2.5Fin 2.5Unf 2Story SFoyer   SLvl 
  0.10   0.01   0.50   0.01   0.01   0.30   0.03   0.04 

$OverallQual
x
   1    2    3    4    5    6    7    8    9   10 
0.00 0.00 0.01 0.08 0.27 0.26 0.22 0.12 0.03 0.01 

$OverallCond
x
   1    2    3    4    5    6    7    8    9 
0.00 0.00 0.02 0.04 0.56 0.17 0.14 0.05 0.02 

$RoofStyle
x
   Flat   Gable Gambrel     Hip Mansard    Shed 
   0.01    0.78    0.01    0.19    0.00    0.00 

$Exterior1st
x
AsbShng AsphShn BrkComm BrkFace  CBlock CemntBd HdBoard ImStucc MetalSd Plywood   Stone  Stucco VinylSd 
   0.01    0.00    0.00    0.03    0.00    0.04    0.15    0.00    0.15    0.07    0.00    0.02    0.35 
Wd Sdng WdShing 
   0.14    0.02 

$Exterior2nd
x
AsbShng AsphShn Brk Cmn BrkFace  CBlock CmentBd HdBoard ImStucc MetalSd   Other Plywood   Stone  Stucco 
   0.01    0.00    0.00    0.02    0.00    0.04    0.14    0.01    0.15    0.00    0.10    0.00    0.02 
VinylSd Wd Sdng Wd Shng 
   0.35    0.14    0.03 

$MasVnrType
x
 BrkCmn BrkFace    None   Stone 
   0.01    0.31    0.59    0.09 

$ExterQual
x
  Ex   Fa   Gd   TA 
0.04 0.01 0.33 0.62 

$ExterCond
x
  Ex   Fa   Gd   Po   TA 
0.00 0.02 0.10 0.00 0.88 

$Foundation
x
BrkTil CBlock  PConc   Slab  Stone   Wood 
  0.10   0.43   0.44   0.02   0.00   0.00 

$BsmtQual
x
  Ex   Fa   Gd   TA 
0.08 0.03 0.42 0.47 

$BsmtCond
x
  Fa   Gd   Po   TA 
0.04 0.04 0.00 0.91 

$BsmtExposure
x
  Av   Gd   Mn   No 
0.15 0.09 0.08 0.68 

$BsmtFinType1
x
 ALQ  BLQ  GLQ  LwQ  Rec  Unf 
0.15 0.10 0.29 0.06 0.10 0.30 

$BsmtFinType2
x
 ALQ  BLQ  GLQ  LwQ  Rec  Unf 
0.01 0.02 0.01 0.03 0.04 0.88 

$Heating
x
Floor  GasA  GasW  Grav  OthW  Wall 
 0.00  0.98  0.01  0.00  0.00  0.00 

$HeatingQC
x
  Ex   Fa   Gd   Po   TA 
0.51 0.03 0.17 0.00 0.29 

$CentralAir
x
   N    Y 
0.07 0.93 

$Electrical
x
FuseA FuseF FuseP   Mix SBrkr 
 0.06  0.02  0.00  0.00  0.91 

$BsmtFullBath
x
   0    1    2    3 
0.59 0.40 0.01 0.00 

$BsmtHalfBath
x
   0    1    2 
0.94 0.05 0.00 

$HalfBath
x
   0    1    2 
0.62 0.37 0.01 

$BedroomAbvGr
x
   0    1    2    3    4    5    6    8 
0.00 0.03 0.25 0.55 0.15 0.01 0.00 0.00 

$KitchenAbvGr
x
   0    1    2    3 
0.00 0.95 0.04 0.00 

$KitchenQual
x
  Ex   Fa   Gd   TA 
0.07 0.03 0.40 0.50 

$TotRmsAbvGrd
x
   2    3    4    5    6    7    8    9   10   11   12   13   14   15 
0.00 0.01 0.07 0.19 0.28 0.22 0.13 0.05 0.03 0.01 0.01 0.00 0.00 0.00 

$Functional
x
Maj1 Maj2 Min1 Min2  Mod  Sev  Typ 
0.01 0.00 0.02 0.02 0.01 0.00 0.93 

$Fireplaces
x
   0    1    2    3    4 
0.47 0.45 0.08 0.00 0.00 

$GarageType
x
 2Types  Attchd Basment BuiltIn CarPort  Detchd 
   0.01    0.60    0.01    0.06    0.01    0.31 

$GarageFinish
x
 Fin  RFn  Unf 
0.24 0.29 0.47 

$GarageCars
x
   0    1    2    3    4    5 
0.06 0.25 0.56 0.12 0.00 0.00 

$GarageQual
x
  Ex   Fa   Gd   Po   TA 
0.00 0.07 0.01 0.00 0.92 

$GarageCond
x
  Ex   Fa   Gd   Po   TA 
0.00 0.06 0.01 0.01 0.93 

$PavedDrive
x
   N    P    Y 
0.06 0.02 0.92 

$MoSold
x
   1    2    3    4    5    6    7    8    9   10   11   12 
0.04 0.04 0.07 0.10 0.14 0.17 0.16 0.08 0.04 0.06 0.05 0.04 

$SaleType
x
  COD   Con ConLD ConLI ConLw   CWD   New   Oth    WD 
 0.03  0.00  0.01  0.00  0.00  0.00  0.08  0.00  0.87 

$SaleCondition
x
Abnorml AdjLand  Alloca  Family  Normal Partial 
   0.07    0.00    0.01    0.01    0.82    0.09 

$What.kind.of.data
x
 test.data train.data 
         0          1 

$Street.factor
x
 bad.Street good.Street 
          0           1 

$LotArea.OverallQual.factor
x
   bad.OverallQual   good.overallQual medium.overallQual 
              0.03               0.64               0.33 

$Overallqual.neigh
x
   high.overall.neigh low.overallqual.neigh med.overallqual.neigh 
                 0.20                  0.04                  0.76 

$OverallQual.salecond
x
  high.overall.saleprice    low.overall.saleprice medium.overall.saleprice 
                    0.16                     0.02                     0.83 

$OverallCond.overallqual
x
high.overall.qual  low.overall.qual  med.overall.qual 
             0.58              0.06              0.36 

$TotRmsAbvGrd.saleprice
x
high.sale.totrms  low.sale.totrms  med.sale.totrms 
            0.05             0.54             0.40 

$YearBuilt.as.factor
x
   1    2    3    4    5    6    7 
0.00 0.02 0.09 0.12 0.29 0.23 0.25 

$YearRemodAdd.as.factor
x
   1    2    3    4 
0.25 0.25 0.25 0.25 

$total_bsmt_bath
x
 1.5    2  2.5    3  3.5  4.5 
0.54 0.05 0.40 0.01 0.01 0.00 

Split the data into train and test and feature importances

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars
Registered S3 method overwritten by 'htmlwidgets':
  method           from         
  print.htmlwidget tools:rstudio

We will create various transformations for each training and test data

We will create a lasso model for feature selection

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars
Resampling: cross-validation
Measures:             rmsle     
[Resample] iter 1:    0.1122575 
[Resample] iter 2:    0.1371049 
[Resample] iter 3:    0.1304579 


Aggregated Result: rmsle.test.mean=0.1266068

We will also create a bagging elastic-net model we well

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars
Resampling: cross-validation
Measures:             rmsle     
[Resample] iter 1:    0.1446785 
[Resample] iter 2:    0.1231925 
[Resample] iter 3:    0.1194127 


Aggregated Result: rmsle.test.mean=0.1290946

We will also create a SVM we well

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,GarageCars,TotRmsAbvGrd,Fireplaces
Resampling: cross-validation
Measures:             rmsle     
[Resample] iter 1:    0.1456177 
[Resample] iter 2:    0.1260868 
[Resample] iter 3:    0.1120551 


Aggregated Result: rmsle.test.mean=0.1279199

We will also create a SVM 2 we well. The warning can be safely ignored

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars
Resampling: cross-validation
Measures:             rmsle     
[Resample] iter 1:    0.1442733 
[Resample] iter 2:    0.1266494 
[Resample] iter 3:    0.1101696 


Aggregated Result: rmsle.test.mean=0.1270308


Warning in svm.default(x, y, scale = scale, ..., na.action = na.action) :
  Variable(s) ‘X3SsnPorch’ and ‘PoolArea’ constant. Cannot scale data.
Warning in svm.default(x, y, scale = scale, ..., na.action = na.action) :
  Variable(s) ‘PoolArea’ constant. Cannot scale data.
Warning in svm.default(x, y, scale = scale, ..., na.action = na.action) :
  Variable(s) ‘PoolArea’ constant. Cannot scale data.
Warning in svm.default(x, y, scale = scale, ..., na.action = na.action) :
  Variable(s) ‘PoolArea’ constant. Cannot scale data.

We will also create a KSVM we well

Attaching package: ‘kernlab’

The following object is masked from ‘package:purrr’:

    cross

The following object is masked from ‘package:ggplot2’:

    alpha

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,GarageCars,TotRmsAbvGrd,Fireplaces
Resampling: cross-validation
Measures:             rmsle     
[Resample] iter 1:    0.1323306 
[Resample] iter 2:    0.1286154 
[Resample] iter 3:    0.1267248 


Aggregated Result: rmsle.test.mean=0.1292236

We will also create a XGB we well

Resampling: cross-validation
Measures:             rmsle.test   
[Resample] iter 1:    0.2016001    
[Resample] iter 2:    0.2005690    
[Resample] iter 3:    0.2102933    


Aggregated Result: rmsle.test.mean=0.2041542

[1] 0.1984432

GLMBOOST

Resampling: cross-validation
Measures:             rmsle.test   
[Resample] iter 1:    0.1526286    
[Resample] iter 2:    0.1460310    
[Resample] iter 3:    0.1697131    
[Resample] iter 4:    0.1452754    
[Resample] iter 5:    0.1594117    


Aggregated Result: rmsle.test.mean=0.1546120

[1] 0.149486

We will also create a SVM 3 we well

Stacking: Lasso Model

We will create a LASSO model as our superlearner

The base learners were svm, and ksvm and I used a lasso model as our super-learner

Resampling: cross-validation
Measures:             rmsle.test   
[Resample] iter 1:    0.0719159    
[Resample] iter 2:    0.0263779    
[Resample] iter 3:    0.0563136    
[Resample] iter 4:    0.0316589    
[Resample] iter 5:    0.0419256    


Aggregated Result: rmsle.test.mean=0.0456384

stacked.data.train<-add_column(stacked.data.train , stacked.model = bb.train.stacked)
stacked.data.test<-add_column(stacked.data.test , stacked.model = bb.test.stacked)

Stacking: Benchmarking

set.seed(2)
svm.reg.scale<-makeLearner("regr.svm" , par.vals = list(nu = 0.5, cost = 3 , tolerance = 0.001), id = "svm.scale")
ksvm.reg<-makeLearner("regr.ksvm", par.vals = list(C = 2, epsilon = 0.001, nu = 0.2), id = "ksvm")
svm.reg.2<-makeLearner("regr.svm", par.vals = list(cost = 4, tolerance = 0.005 , nu = 0.5, scale = TRUE) , 
                       id = "svm.reg2")
svm.reg<-makeLearner("regr.svm", par.vals = list(cost = 4 , nu = 0.5 ,tolerance = 0.001 ), id = "svm")


lrns = list(svm.reg, svm.reg.2, svm.reg.scale)
train.task<-makeRegrTask(data = train.data %>% select(feat), target = "SalePrice")

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,GarageCars,TotRmsAbvGrd,Fireplaces

repcv = makeResampleDesc("RepCV" , folds = 3, rep = 5)

b1<-benchmark(lrns, train.task, repcv, models = TRUE , measures = rmsle, show.info = FALSE)

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,GarageCars,TotRmsAbvGrd,Fireplaces

Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars
Warning in makeTask(type = type, data = data, weights = weights, blocking = blocking,  :
  Empty factor levels were dropped for columns: MSSubClass,TotRmsAbvGrd,Fireplaces,GarageCars

Stacking: Benchmarking boxplots

We see that benchmarking averaged the results of all the ML models

We can create another stacking model to increase performance, or we can manually adjust the weights to see how well we do it.

No id variables; using all as measure variables

Stacking: Manuel weights based on RMSLE

It is very evident that the custom weights we have created gave us the greatest overall performance. We will use these custom weights for represent our test data as well.

No id variables; using all as measure variables

The final predictions

z<-stacked.data.test %>% select(c("svm","ksvm",  "glm.boost", "stacked.model"))

a<-0.6
b<-0
c<-0.2
d<-0.7
pp<-apply(z , 1, function(x){return(weighted.mean(x, w = c(a,b,c,d)))})
print(head(pp))

       1        2        3        4        5        6 
122418.3 157235.5 192111.2 195803.6 186949.0 173801.4